Skip to content

[SC-8072] Support threshold lines in unit metric plots#293

Merged
juanmleng merged 3 commits intomainfrom
juan5508/sc-8072/support-threshold-lines-in-unit-metric-plots
Jan 15, 2025
Merged

[SC-8072] Support threshold lines in unit metric plots#293
juanmleng merged 3 commits intomainfrom
juan5508/sc-8072/support-threshold-lines-in-unit-metric-plots

Conversation

@juanmleng
Copy link
Contributor

@juanmleng juanmleng commented Jan 14, 2025 •
edited
Loading

Internal Notes for Reviewers

In addition to updates in the ValidMind library, supporting thresholds in log_metric() requires changes in both the backend and frontend repositories. The following pull requests have been created for these repos:

External Release Notes

Description
Enhances the log_metric() function to accept multiple named thresholds as a dictionary input. This allows users to define and track multiple threshold levels (e.g., "high_risk": 0.6, "medium_risk": 0.7, "low_risk": 0.8) for each unit metric.

Changes

  • Added threshold support in log_metric()
  • Added notebook how_to\log_metrics_over_time.ipynb

Testing

  • Tested with various threshold configurations
  • Verified threshold line rendering in the UI
  • Validated threshold updates reflect immediately in visualization

Example

log_metric(
    key="AUC Score",
    value=auc,
    recorded_at=datetime(2024, 1, 1),
    thresholds={
        "high_risk": 0.6,
        "medium_risk": 0.7,
        "low_risk": 0.8,
    }
)
log_metric_auc_4

@juanmleng juanmleng added the enhancement New feature or request label Jan 14, 2025
@juanmleng juanmleng self-assigned this Jan 14, 2025
johnwalz97
johnwalz97 approved these changes Jan 14, 2025
View reviewed changes
Copy link
Contributor

@johnwalz97 johnwalz97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@github-actions
Copy link
Contributor

github-actions bot commented Jan 15, 2025

PR Summary

This pull request introduces enhancements to the ValidMind library by adding support for logging thresholds alongside metrics. The changes are primarily focused on the log_metric and alog_metric functions within the api_client.py file. These functions now accept an optional thresholds parameter, which allows users to specify threshold values for metrics. This feature is useful for visualizing metrics over time and identifying potential issues by setting reference lines in metric visualizations. The notebook log_metrics_over_time.ipynb has been updated to demonstrate how to use this new feature, including examples of logging multiple metrics with custom thresholds.

Test Suggestions

  • Test logging a metric with valid threshold values to ensure they are correctly processed and stored.
  • Test logging a metric without thresholds to verify that the function handles it gracefully.
  • Test logging a metric with invalid threshold data types (e.g., a list instead of a dictionary) to ensure proper error handling.
  • Verify that the notebook log_metrics_over_time.ipynb runs without errors and correctly logs metrics with thresholds.
  • Check the visualization of metrics over time to ensure thresholds are displayed as expected.

@juanmleng juanmleng merged commit dc5c4ca into main Jan 15, 2025
6 checks passed
@johnwalz97 johnwalz97 deleted the juan5508/sc-8072/support-threshold-lines-in-unit-metric-plots branch February 27, 2025 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@johnwalz97 johnwalz97 johnwalz97 approved these changes

@cachafla cachafla Awaiting requested review from cachafla

Assignees

@juanmleng juanmleng

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@juanmleng @johnwalz97